Skip to content

feat: Add Chat Generator supporting OpenAI Responses API #9808

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 35 commits into
base: main
Choose a base branch
from
Open

feat: Add Chat Generator supporting OpenAI Responses API #9808

wants to merge 35 commits into from
+1,998 −2

Conversation

@Amnah199
Copy link
Contributor

@Amnah199 Amnah199 commented Sep 22, 2025 •
edited
Loading

Related Issues

Proposed Changes:

  • Add a chat generator that uses openai's responses endpoint.
  • The tools param in init and run methods allows openai and MCP tools besides haystack Tools.
  • Structured outputs can be enabled by passing text_format in generation_kwargs.
  • Reasoning can be enabled by setting reasoning params in generation_kwargs.
  • There are other features in Responses API like background requests etc, which are not supported in this iteration.

How did you test it?

  • Add new unit tests to test features.
  • Add new integration tests.

Notes for the reviewer

Using OpenAI tools and MCP tools doesn’t behave like standard function calls.

  • Tool calls are usually identified by type="function_call". However, for OpenAI/MCP tools, the type value varies depending on the tool being used — for example, type="web_search_call".
  • Currently, we only return standard ResponseFunctionToolCall objects. As a result, tool calls from OpenAI/MCP tools won’t appear in the ChatMessage, so the user won’t see them.
  • This is fine, since these tool calls are handled internally by OpenAI to generate the final response. The user already receives the output.

Checklist

  • I have read the contributors guidelines and the code of conduct
  • I have updated the related issue with new insights and changes
  • I added unit tests and updated the docstrings
  • I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test: and added ! in case the PR includes breaking changes.
  • I documented my code
  • I ran pre-commit hooks and fixed any issue

@github-actions github-actions bot added the type:documentation Improvements on the docs label Sep 22, 2025
@coveralls
Copy link
Collaborator

coveralls commented Sep 22, 2025 •
edited
Loading

Pull Request Test Coverage Report for Build 18779147146

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 0 changed or added relevant lines in 0 files are covered.
  • 1 unchanged line in 1 file lost coverage.
  • Overall coverage decreased (-0.9%) to 91.313%
Files with Coverage Reduction New Missed Lines %
components/builders/answer_builder.py 1 98.57%
Totals Coverage Status
Change from base Build 18690275685: -0.9%
Covered Lines: 13581
Relevant Lines: 14873
💛 - Coveralls

@Amnah199 Amnah199 requested a review from vblagoje October 13, 2025 07:57
@Amnah199 Amnah199 marked this pull request as ready for review October 13, 2025 10:15
@Amnah199 Amnah199 requested a review from a team as a code owner October 13, 2025 10:15
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
sjrl
sjrl reviewed Oct 14, 2025
View reviewed changes
Amnah199
Amnah199 commented Oct 15, 2025
View reviewed changes
@vercel
Copy link

vercel bot commented Oct 22, 2025 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
haystack-docs Ready Ready Preview Comment Oct 31, 2025 2:17pm

Amnah199
Amnah199 commented Oct 23, 2025
View reviewed changes
@Amnah199 Amnah199 marked this pull request as ready for review October 24, 2025 09:10
vblagoje
vblagoje reviewed Oct 24, 2025
View reviewed changes
Copy link
Member

@vblagoje vblagoje left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Amnah199 left some initial comments; didn't go into details yet

releasenotes/notes/add-openai-responses-chatgenerator-52ca7457a4e61db1.yaml
Added the OpenAIResponsesChatGenerator, a new component that integrates OpenAI's Responses API into Haystack.
This unlocks several advanced capabilities from the Responses API:
- Allowing retrieval of concise summaries of the model's reasoning process.
- Allowing the use of native OpenAI or MCP tool formats, along with Haystack Tool objects and Toolset instances.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What else do we support and integrate well from our ecosystem into Responses?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Structured outputs could be one.

haystack/components/generators/chat/openai_responses.py
Completes chats using OpenAI's Responses API.
It works with the gpt-4 and o-series models and supports streaming responses
from OpenAI API. It uses [ChatMessage](https://docs.haystack.deepset.ai/docs/chatmessage)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd mention here, including the o3-deep-research,
o4-mini-deep-research, and gpt-5-codex among others. People don't know this in general that you can use these specialized model/agents with this CG. Or do we don't want to mention that at this time, because they need to run usually in background etc

haystack/components/generators/chat/openai_responses.py
OpenAI API. Use the `**generation_kwargs` argument when you initialize
the component or when you run it. Any parameter that works with
`openai.Responses.create` will work here too.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we mention somewhere here what we don't support i.e background tasks etc

@Amnah199 Amnah199 requested a review from anakin87 October 28, 2025 16:50
@anakin87
Copy link
Member

Please merge main to update tests coverage information in #9808 (comment). Let's see if more unit tests are needed.

@sjrl sjrl mentioned this pull request Oct 31, 2025
Open
@anakin87
Copy link
Member

We discussed with @Amnah199 about Function tool call call_id and id. In short, we need to store both.

We considered two possible solutions:

  • Store one of them in ToolCall.id and use ChatMessage.meta to keep the other (or a mapping between them). Since a single ChatMessage can contain multiple tool calls, this approach isn't straightforward but might work.

  • If the previous solution is too complex, add an extra dictionary field to ToolCall (and ToolCallDelta) to store additional info (e.g. provider-specific info). This seems preferable to introducing a dedicated call_id field, which could be confusing for users, and it also allows for future extensibility.

Since I'll be on PTO next week, feel free to merge this PR when it's ready.

@Amnah199
Copy link
Contributor Author

Amnah199 commented Oct 31, 2025 •
edited
Loading

@sjrl Would love your input here.
If we plan to evetually add an extra field to ToolCall, I would implement it that way now as it makes the task much simpler overall.

However, if that’s unlikely (or a definite no), I can try to make the first option work instead.

Context: Here’s the commit showing what the first approach might look like.
On top of that, we’d need to handle lists in convert_messages_to_responses_api_format(messages: list[ChatMessage], …) (as suggested by @anakin87) to correctly map call_id to ToolCallResult as the ChatMessage.meta containing tool results dont have this mapping.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vblagoje vblagoje vblagoje left review comments

@sjrl sjrl sjrl left review comments

@anakin87 anakin87 Awaiting requested review from anakin87

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

topic:build/distribution topic:tests type:documentation Improvements on the docs

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add support for OpenAI's Responses API

6 participants

@Amnah199 @coveralls @anakin87 @vblagoje @sjrl